CS224N Final Project Abstract Meta-Concept Features for Text-Illustration

Cross language-image retrieval is a problem of high interest that is at the frontier between computer vision and natural language processing. State-of-the-art methods learn a common space with regard to some constraints of correlation or similarity from two textual and visual modalities that are processed in parallel and possibly jointly. This paper proposes a different approach that considers the cross-modal problem as a supervised mapping of visual modalities to textual ones. Each modality is thus seen as a particular projection of an abstract meta-concept. In practice, this space is learned through an asymmetric process, where the textual modality is used to generate a multi-label representation, further used to map the visual modality through a simple multi-layer perceptron. While being quite easy to implement, the experiments show that our approach significantly outperforms the state-of-the-art on FlickR-8K and FlickR-30K datasets for the text-illustration task.